[DRBD-user] Issues migrating from drbdmanage to linstor with linstor 0.9.4
Wolfgang Walkowiak
walkowiak at hep.physik.uni-siegen.de
Sun Mar 31 19:32:05 CEST 2019
Hi,
another bit of information:
Following the same steps as before, after issuing
monetbase2:~# linstor storage-pool create lvmthin munchbase2 drbdpool
there is no port assigned:
monetbase2:~# linstor rd l
╭───────────────────────────────╮
┊ ResourceName ┊ Port ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-1802-disk-2 ┊ ┊ ok ┊
╰───────────────────────────────╯
Also, after issuing
monetbase2:~# linstor volume-definition create --vlmnr 0 --minor 125 vm-1802-disk-2 44040192K
there is no VolumeMinor number assigned:
monetbase2:/var/lib/drbd.d# linstor vd l
╭──────────────────────────────────────────────────────────╮
┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ State ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ vm-1802-disk-2 ┊ 0 ┊ ┊ 42 GiB ┊ ok ┊
╰──────────────────────────────────────────────────────────╯
I'd have expected port 7014 and VolumeMinor 125 to be assigned.
Maybe this helps for finding the cause.
Thanks again,
w.w.
On Sun, 31 Mar 2019, Wolfgang Walkowiak wrote:
>
> Hi,
>
> we are experiencing issues while we try to migrate an existing DRBD cluster
> from drbdmanage to linstor (Debian stable with proxmox for VMs) following the
> instructions at
> https://docs.linbit.com/docs/users-guide-9.0/#s-linstor-migrate-from-dm
>
> The issue is that the resource files created when I execute the migration
> script (dmmmigrate.sh in my case) created by
>
> monetbase2:~# drbdmanage export-ctrlvol > ctrlvol.json
> monetbase2:~# linstor dm-migrate ctrlvol.json dmmmigrate.sh
>
> contain wrong port and wrong VolumeMinor numbers.
> However, the numbers are correctly assigned in the lines for the resource and
> volume creation inside dmmigrate.sh. This is somewhat
> puzzeling.
>
> In order to debug it, I decided to manually execute the contents of
> dmmmigrate.sh line by line. (After reverting to drbdmanage and a clean
> linstor installation.) I took the volume vm-1802-disk-2
> which according to [A] (see below) uses port 7014 and VolumeMinor 125.
>
> First I made sure that 'drbdmanage shutdown -qc' worked and DBUS wouldn't
> restart it (on both DRBD nodes). Then I disabled /etc/drbd.d/drbdctrl.res
> and /etc/drbd.d/drbdmanage-resources.res on both nodes, started
> linstor-satellite on them and the linstor-controller on node monetbase2.
>
> Then, following [B] from dmmmigrate.sh below, on the controller node,
> monetbase2, I executed:
>
> monetbase2:~# LS_CONTROLLERS="localhost"
> monetbase2:~# export LS_CONTROLLERS
> monetbase2:~# linstor node create --node-type Combined munchbase2 10.0.7.13
> monetbase2:~# linstor node create --node-type Combined monetbase2 10.0.7.12
> monetbase2:~# linstor n l
> monetbase2:~# linstor storage-pool-definition create drbdpool
> monetbase2:~# linstor storage-pool create lvmthin munchbase2 drbdpool
> drbdpool/drbdthinpool
> monetbase2:~# linstor storage-pool create lvmthin monetbase2 drbdpool
> drbdpool/drbdthinpool
> monetbase2:~# linstor sp l
>
> The created storage pool looked fine.
>
> Going on:
>
> monetbase2:~# linstor resource-definition create --port 7014 vm-1802-disk-2
> monetbase2:~# linstor resource-definition drbd-options --allow-two-primaries
> yes vm-1802-disk-2
> monetbase2:~# linstor volume-definition create --vlmnr 0 --minor 125
> vm-1802-disk-2 44040192K
> monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
> OverrideVlmId vm-1802-disk-2_00
> monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0
> AllowLargerVolumeSize true
> monetbase2:~# linstor resource create --node-id 1 --storage-pool drbdpool
> munchbase2 vm-1802-disk-2
> monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
> monetbase2 vm-1802-disk-2
>
> the last command produced an error message:
>
> monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool
> monetbase2 vm-1802-disk-2
> SUCCESS:
> Description:
> New resource 'vm-1802-disk-2' on node 'monetbase2' registered.
> Details:
> Resource 'vm-1802-disk-2' on node 'monetbase2' UUID is:
> 30b98a7f-cb31-4635-89cd-6860b13fa5cc
> SUCCESS:
> Description:
> Volume with number '0' on resource 'vm-1802-disk-2' on node 'monetbase2'
> successfully registered
> Details:
> Volume UUID is: 86fc67d7-18f1-41b1-9718-4b15ddf894f9
> ERROR:
> (Node: 'monetbase2') Failed to adjust DRBD resource vm-1802-disk-2
> Show reports:
> linstor error-reports show 5CA090AF-97F26-000000
> ERROR:
> (Node: 'munchbase2') Failed to adjust DRBD resource vm-1802-disk-2
> Show reports:
> linstor error-reports show 5CA09099-17F52-000000
>
> Now, looking at e.g. the first error report (see [C] and [D] below for the
> full error reports), I see:
>
> The full command line executed was:
> drbdadm -vvv adjust vm-1802-disk-2
>
> The external command sent the following output data:
> drbdsetup detach --diskless 125
> drbdsetup del-minor 125
> drbdsetup new-minor vm-1802-disk-2 1000 0
> drbdsetup disconnect vm-1802-disk-2 1
> drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
> ipv4:10.0.7.13:7014
> drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
> ipv4:10.0.7.13:7000
>
> that a wrong port and a wrong VolumeMinor number are forced.
>
> Looking into /var/lib/linstor.d/vm-1802-disk-2.res (see [E] below),
> I see that the wrong asignments are contained in this resource file.
>
> I also see the wrong assignments using linstor now:
>
> monetbase2:/var/lib/linstor.d# linstor vd l -p
> +----------------------------------------------------------+
> | ResourceName | VolumeNr | VolumeMinor | Size | State |
> |----------------------------------------------------------|
> | vm-1802-disk-2 | 0 | 1000 | 42 GiB | ok |
> +----------------------------------------------------------+
> monetbase2:/var/lib/linstor.d# linstor r l -p
> +--------------------------------------------------------+
> | ResourceName | Node | Port | Usage | State |
> |--------------------------------------------------------|
> | vm-1802-disk-2 | monetbase2 | 7000 | Unused | Diskless |
> | vm-1802-disk-2 | munchbase2 | 7000 | Unused | Diskless |
> +--------------------------------------------------------+
>
> Unfortunately, this spoils my attempt to migrate from drbdmanage to
> linstor on an active system since I could not figure out,
> how to modify the port number and the VolumeMinor using the linstor client.
> I'll revert to the drbdmanage setup for now.
>
> However, I would be grateful if someone knowledgable could have a look
> and see whether this is a bug or a mistake on my side. A workaround
> (like how force these two numbers with a linstor client command) would be
> appreciated.
>
> Sorry for the long post. Hopefully I have provided enough details.
>
> Thank you for having a look,
>
> w.w.
>
>
> [A] -------------------------
>
> munchbase2:~# drbdmanage list-volumes -p -g Port
> +----------------------------------------------------------------------------+
> | Name | Vol ID | Size | Minor | Port | | State
> |
> |----------------------------------------------------------------------------|
> | vm-1801-disk-1 | 0 | 32 GiB | 103 | 7000 | | ok |
> | vm-103-disk-2 | 0 | 1 TiB | 104 | 7001 | | ok |
> | vm-103-disk-3 | 0 | 2 TiB | 105 | 7002 | | ok |
> | vm-103-disk-4 | 0 | 1 TiB | 106 | 7003 | | ok |
> | vm-103-disk-5 | 0 | 500 GiB | 107 | 7004 | | ok |
> | vm-102-disk-2 | 0 | 1 TiB | 108 | 7005 | | ok |
> | vm-102-disk-3 | 0 | 2 TiB | 109 | 7006 | | ok |
> | vm-102-disk-4 | 0 | 1 TiB | 111 | 7007 | | ok |
> | vm-102-disk-5 | 0 | 500 GiB | 112 | 7008 | | ok |
> | vm-102-disk-6 | 0 | 100 GiB | 113 | 7009 | | ok |
> | vm-1802-disk-1 | 0 | 32 GiB | 114 | 7010 | | ok |
> | vm-105-disk-2 | 0 | 200 GiB | 120 | 7011 | | ok |
> | vm-104-disk-1 | 0 | 18 GiB | 119 | 7012 | | ok |
> | vm-105-disk-1 | 0 | 32 GiB | 121 | 7013 | | ok |
> | vm-1802-disk-2 | 0 | 42 GiB | 125 | 7014 | | ok |
> | vm-1801-disk-12 | 0 | 32 GiB | 123 | 7015 | | ok |
> | vm-103-disk-1 | 0 | 50 GiB | 126 | 7016 | | ok |
> | vm-102-disk-1 | 0 | 50 GiB | 127 | 7017 | | ok |
> | vm-106-disk-1 | 0 | 32 GiB | 128 | 7018 | | ok |
> +----------------------------------------------------------------------------+
>
> [B] -------------------------
>
> dmmmigrate.sh:
>
> ...
> # If the controller is not executed on the local host, set this variable:
> LS_CONTROLLERS="localhost"
> export LS_CONTROLLERS
> ...
> ### Nodes ###
> linstor node create --node-type Combined munchbase2 10.0.7.13
> linstor node create --node-type Combined monetbase2 10.0.7.12
>
> ### Storage ###
> linstor storage-pool-definition create drbdpool
> linstor storage-pool create lvmthin munchbase2 drbdpool drbdpool/drbdthinpool
> linstor storage-pool create lvmthin monetbase2 drbdpool drbdpool/drbdthinpool
> ...
> ### Resource: vm-1802-disk-2 ###
> linstor resource-definition create --port 7014 vm-1802-disk-2
> linstor resource-definition drbd-options --allow-two-primaries yes
> vm-1802-disk-2
> linstor volume-definition create --vlmnr 0 --minor 125 vm-1802-disk-2
> 44040192K
> linstor volume-definition set-property vm-1802-disk-2 0 OverrideVlmId
> vm-1802-disk-2_00
> linstor volume-definition set-property vm-1802-disk-2 0 AllowLargerVolumeSize
> true
> linstor resource create --node-id 1 --storage-pool drbdpool munchbase2
> vm-1802-disk-2
> linstor resource create --node-id 0 --storage-pool drbdpool monetbase2
> vm-1802-disk-2
>
>
> [C] ------------------
>
> monetbase2:/etc/linstor# linstor error-reports show 5CA090AF-97F26-000000
> ERROR REPORT 5CA090AF-97F26-000000
>
> ============================================================
>
> Application: LINBIT® LINSTOR
> Module: Satellite
> Version: 0.9.4
> Build ID: ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
> Build time: 2019-03-29T09:40:59+00:00
> Error time: 2019-03-31 12:49:17
> Node: monetbase2
>
> ============================================================
>
> Reported error:
> ===============
>
> Description:
> Failed to adjust DRBD resource vm-1802-disk-2
>
> Category: LinStorException
> Class name: ResourceException
> Class canonical name:
> com.linbit.linstor.storage.layer.exceptions.ResourceException
> Generated at: Method 'adjustDrbd', Source file
> 'DrbdLayer.java', Line #432
>
> Error message: Failed to adjust DRBD resource
> vm-1802-disk-2
>
> Error context:
> An error occurred while processing resource 'Node: 'monetbase2', Rsc:
> 'vm-1802-disk-2''
>
> Call backtrace:
>
> Method Native Class:Line number
> adjustDrbd N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
> process N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
> process N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
> processResourcesAndTheirSnapshots N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
> phaseDispatchDeviceHandlers N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
> devMgrLoop N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
> run N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
> run N java.lang.Thread:748
>
> Caused by:
> ==========
>
> Description:
> Execution of the external command 'drbdadm' failed.
> Cause:
> The external command exited with error code 1.
> Correction:
> - Check whether the external program is operating properly.
> - Check whether the command line is correct.
> Contact a system administrator or a developer if the command line is no
> longer valid
> for the installed version of the external program.
> Additional information:
> The full command line executed was:
> drbdadm -vvv adjust vm-1802-disk-2
>
> The external command sent the following output data:
> drbdsetup detach --diskless 125
> drbdsetup del-minor 125
> drbdsetup new-minor vm-1802-disk-2 1000 0
> drbdsetup disconnect vm-1802-disk-2 1
> drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014
> ipv4:10.0.7.13:7014
> drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
> ipv4:10.0.7.13:7000
>
>
> The external command sent the follwing error information:
> [ne] vol:0 minor differs: r=125 c=1000
> Value of 'shared-secret' differs: r=bXj5Gn1qO2YXgQoX6wLE
> c=aPDz2JoXsWxbscd2j70f
> vm-1802-disk-2: Failure: (173) Combination of local address(port) and
> remote address(port) already in use
>
> Command 'drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000
> ipv4:10.0.7.13:7000' terminated with exit code 10
>
>
> Category: LinStorException
> Class name: ExtCmdFailedException
> Class canonical name: com.linbit.extproc.ExtCmdFailedException
> Generated at: Method 'execute', Source file
> 'DrbdAdm.java', Line #455
>
> Error message: The external command 'drbdadm' exited
> with error code 1
>
>
> Call backtrace:
>
> Method Native Class:Line number
> execute N
> com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
> adjust N
> com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
> adjustDrbd N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
> process N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
> process N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
> processResourcesAndTheirSnapshots N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
> phaseDispatchDeviceHandlers N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
> devMgrLoop N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
> run N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
> run N java.lang.Thread:748
>
>
> END OF ERROR REPORT.
>
> [D] ------------------
>
> monetbase2:/etc/linstor# linstor error-reports show 5CA09099-17F52-000000
> ERROR REPORT 5CA09099-17F52-000000
>
> ============================================================
>
> Application: LINBIT® LINSTOR
> Module: Satellite
> Version: 0.9.4
> Build ID: ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
> Build time: 2019-03-29T09:40:59+00:00
> Error time: 2019-03-31 12:49:17
> Node: munchbase2
>
> ============================================================
>
> Reported error:
> ===============
>
> Description:
> Failed to adjust DRBD resource vm-1802-disk-2
>
> Category: LinStorException
> Class name: ResourceException
> Class canonical name:
> com.linbit.linstor.storage.layer.exceptions.ResourceException
> Generated at: Method 'adjustDrbd', Source file
> 'DrbdLayer.java', Line #432
>
> Error message: Failed to adjust DRBD resource
> vm-1802-disk-2
>
> Error context:
> An error occurred while processing resource 'Node: 'munchbase2', Rsc:
> 'vm-1802-disk-2''
>
> Call backtrace:
>
> Method Native Class:Line number
> adjustDrbd N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
> process N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
> process N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
> processResourcesAndTheirSnapshots N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
> phaseDispatchDeviceHandlers N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
> devMgrLoop N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
> run N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
> run N java.lang.Thread:748
>
> Caused by:
> ==========
>
> Description:
> Execution of the external command 'drbdadm' failed.
> Cause:
> The external command exited with error code 1.
> Correction:
> - Check whether the external program is operating properly.
> - Check whether the command line is correct.
> Contact a system administrator or a developer if the command line is no
> longer valid
> for the installed version of the external program.
> Additional information:
> The full command line executed was:
> drbdadm -vvv adjust vm-1802-disk-2
>
> The external command sent the following output data:
> drbdsetup new-peer vm-1802-disk-2 0 --_name=monetbase2
> --allow-two-primaries=yes --shared-secret=aPDz2JoXsWxbscd2j70f
> --cram-hmac-alg=sha1
> drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
> ipv4:10.0.7.12:7000
>
>
> The external command sent the follwing error information:
> Value of 'rs-discard-granularity' differs: r=524288 c=65536
> vm-1802-disk-2: Failure: (173) Combination of local address(port) and
> remote address(port) already in use
>
> Command 'drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000
> ipv4:10.0.7.12:7000' terminated with exit code 10
>
>
> Category: LinStorException
> Class name: ExtCmdFailedException
> Class canonical name: com.linbit.extproc.ExtCmdFailedException
> Generated at: Method 'execute', Source file
> 'DrbdAdm.java', Line #455
>
> Error message: The external command 'drbdadm' exited
> with error code 1
>
>
> Call backtrace:
>
> Method Native Class:Line number
> execute N
> com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
> adjust N
> com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
> adjustDrbd N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
> process N
> com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
> process N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
> processResourcesAndTheirSnapshots N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
> dispatchResources N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
> phaseDispatchDeviceHandlers N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
> devMgrLoop N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
> run N
> com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
> run N java.lang.Thread:748
>
>
> END OF ERROR REPORT.
>
> [E] --------------------------
> /var/lib/linstor.d/vm-1802-disk-2.res:
>
> # This file was generated by linstor(0.9.4), do not edit manually.
>
> resource "vm-1802-disk-2"
> {
> template-file "linstor_common.conf";
>
> net
> {
> cram-hmac-alg sha1;
> shared-secret "TKoHhL4KThL165NGg/ja";
> allow-two-primaries yes;
> }
>
> on monetbase2
> {
> volume 0
> {
> disk /dev/drbdpool/vm-1802-disk-2_00;
> disk
> {
> discard-zeroes-if-aligned yes;
> rs-discard-granularity 65536;
> }
> meta-disk internal;
> device minor 1000;
> }
> node-id 0;
> }
>
> on munchbase2
> {
> volume 0
> {
> disk /dev/drbd/this/is/not/used;
> disk
> {
> discard-zeroes-if-aligned yes;
> rs-discard-granularity 65536;
> }
> meta-disk internal;
> device minor 1000;
> }
> node-id 1;
> }
>
> connection
> {
> host monetbase2 address ipv4 10.0.7.12:7000;
> host munchbase2 address ipv4 10.0.7.13:7000;
> }
> }
>
>
>
> ==========================================================2019-03-31==13:56==
>
> Dr. Wolfgang Walkowiak Phone: +49-271-740-3889
> Fakultaet IV / Physik Fax : +49-271-740-3886
> Emmy Noether Campus
> Universitaet Siegen --> Wolfgang.Walkowiak at hep.physik.uni-siegen.de
> Walter-Flex-Str. 3 Wolfgang.Walkowiak at gmx.net
> 57068 Siegen
> Germany
>
> =============================================================================
==========================================================2019-03-31==19:27==
Dr. Wolfgang Walkowiak Phone: +49-271-740-3889
Fakultaet IV / Physik Fax : +49-271-740-3886
Emmy Noether Campus
Universitaet Siegen --> Wolfgang.Walkowiak at hep.physik.uni-siegen.de
Walter-Flex-Str. 3 Wolfgang.Walkowiak at gmx.net
57068 Siegen
Germany
=============================================================================
More information about the drbd-user
mailing list