[DRBD-user] Issues migrating from drbdmanage to linstor with linstor 0.9.4

Wolfgang Walkowiak walkowiak at hep.physik.uni-siegen.de
Sun Mar 31 14:29:22 CEST 2019


Hi,

we are experiencing issues while we try to migrate an existing DRBD 
cluster from drbdmanage to linstor (Debian stable with proxmox for VMs) 
following the instructions at
https://docs.linbit.com/docs/users-guide-9.0/#s-linstor-migrate-from-dm

The issue is that the resource files created when I execute the 
migration script (dmmmigrate.sh in my case) created by

monetbase2:~# drbdmanage export-ctrlvol > ctrlvol.json
monetbase2:~# linstor dm-migrate ctrlvol.json dmmmigrate.sh

contain wrong port and wrong VolumeMinor numbers.
However, the numbers are correctly assigned in the lines for the 
resource and volume creation inside dmmigrate.sh.  This is somewhat
puzzeling.

In order to debug it, I decided to manually execute the contents of
dmmmigrate.sh line by line.  (After reverting to drbdmanage and a clean 
linstor installation.)  I took the volume vm-1802-disk-2
which according to [A] (see below) uses port 7014 and VolumeMinor 125.

First I made sure that 'drbdmanage shutdown -qc' worked and DBUS wouldn't 
restart it (on both DRBD nodes).  Then I disabled /etc/drbd.d/drbdctrl.res 
and /etc/drbd.d/drbdmanage-resources.res on both nodes, started
linstor-satellite on them and the linstor-controller on node monetbase2.

Then, following [B] from dmmmigrate.sh below, on the controller node,
monetbase2, I executed:

monetbase2:~# LS_CONTROLLERS="localhost"
monetbase2:~# export LS_CONTROLLERS
monetbase2:~# linstor node create --node-type Combined munchbase2 
10.0.7.13
monetbase2:~# linstor node create --node-type Combined monetbase2 
10.0.7.12
monetbase2:~# linstor n l
monetbase2:~# linstor storage-pool-definition create drbdpool
monetbase2:~# linstor storage-pool create lvmthin munchbase2 drbdpool 
drbdpool/drbdthinpool
monetbase2:~# linstor storage-pool create lvmthin monetbase2 drbdpool 
drbdpool/drbdthinpool
monetbase2:~# linstor sp l

The created storage pool looked fine.

Going on:

monetbase2:~# linstor resource-definition create --port 7014 
vm-1802-disk-2
monetbase2:~# linstor resource-definition drbd-options 
--allow-two-primaries yes vm-1802-disk-2
monetbase2:~# linstor volume-definition create --vlmnr 0 --minor 125 
vm-1802-disk-2 44040192K
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0 
OverrideVlmId vm-1802-disk-2_00
monetbase2:~# linstor volume-definition set-property vm-1802-disk-2 0 
AllowLargerVolumeSize true
monetbase2:~# linstor resource create --node-id 1 --storage-pool drbdpool 
munchbase2 vm-1802-disk-2
monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool 
monetbase2 vm-1802-disk-2

the last command produced an error message:

monetbase2:~# linstor resource create --node-id 0 --storage-pool drbdpool 
monetbase2 vm-1802-disk-2
SUCCESS:
Description:
     New resource 'vm-1802-disk-2' on node 'monetbase2' registered.
Details:
     Resource 'vm-1802-disk-2' on node 'monetbase2' UUID is: 
30b98a7f-cb31-4635-89cd-6860b13fa5cc
SUCCESS:
Description:
     Volume with number '0' on resource 'vm-1802-disk-2' on node 
'monetbase2' successfully registered
Details:
     Volume UUID is: 86fc67d7-18f1-41b1-9718-4b15ddf894f9
ERROR:
     (Node: 'monetbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
     linstor error-reports show 5CA090AF-97F26-000000
ERROR:
     (Node: 'munchbase2') Failed to adjust DRBD resource vm-1802-disk-2
Show reports:
     linstor error-reports show 5CA09099-17F52-000000

Now, looking at e.g. the first error report (see [C] and [D] below for the 
full error reports), I see:

     The full command line executed was:
     drbdadm -vvv adjust vm-1802-disk-2

     The external command sent the following output data:
     drbdsetup detach --diskless 125
     drbdsetup del-minor 125
     drbdsetup new-minor vm-1802-disk-2 1000 0
     drbdsetup disconnect vm-1802-disk-2 1
     drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014 
ipv4:10.0.7.13:7014
     drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000 
ipv4:10.0.7.13:7000

that a wrong port and a wrong VolumeMinor number are forced.

Looking into /var/lib/linstor.d/vm-1802-disk-2.res (see [E] below),
I see that the wrong asignments are contained in this resource file.

I also see the wrong assignments using linstor now:

monetbase2:/var/lib/linstor.d# linstor vd l -p
+----------------------------------------------------------+
| ResourceName   | VolumeNr | VolumeMinor | Size   | State |
|----------------------------------------------------------|
| vm-1802-disk-2 | 0        | 1000        | 42 GiB | ok    |
+----------------------------------------------------------+
monetbase2:/var/lib/linstor.d# linstor r l -p
+--------------------------------------------------------+
| ResourceName   | Node       | Port | Usage  |    State |
|--------------------------------------------------------|
| vm-1802-disk-2 | monetbase2 | 7000 | Unused | Diskless |
| vm-1802-disk-2 | munchbase2 | 7000 | Unused | Diskless |
+--------------------------------------------------------+

Unfortunately, this spoils my attempt to migrate from drbdmanage to
linstor on an active system since  I could not figure out,
how to modify the port number and the VolumeMinor using the linstor 
client.  I'll revert to the drbdmanage setup for now.

However, I would be grateful if someone knowledgable could have a look
and see whether this is a bug or a mistake on my side.  A workaround
(like how force these two numbers with a linstor client command) 
would be appreciated.

Sorry for the long post.  Hopefully I have provided enough details.

Thank you for having a look,

w.w.


[A] -------------------------

munchbase2:~# drbdmanage list-volumes -p -g Port
+----------------------------------------------------------------------------+
| Name            | Vol ID |    Size | Minor | Port |                | 
State |
|----------------------------------------------------------------------------|
| vm-1801-disk-1  |      0 |  32 GiB |   103 | 7000 |                | 
ok |
| vm-103-disk-2   |      0 |   1 TiB |   104 | 7001 |                | 
ok |
| vm-103-disk-3   |      0 |   2 TiB |   105 | 7002 |                | 
ok |
| vm-103-disk-4   |      0 |   1 TiB |   106 | 7003 |                | 
ok |
| vm-103-disk-5   |      0 | 500 GiB |   107 | 7004 |                | 
ok |
| vm-102-disk-2   |      0 |   1 TiB |   108 | 7005 |                | 
ok |
| vm-102-disk-3   |      0 |   2 TiB |   109 | 7006 |                | 
ok |
| vm-102-disk-4   |      0 |   1 TiB |   111 | 7007 |                | 
ok |
| vm-102-disk-5   |      0 | 500 GiB |   112 | 7008 |                | 
ok |
| vm-102-disk-6   |      0 | 100 GiB |   113 | 7009 |                | 
ok |
| vm-1802-disk-1  |      0 |  32 GiB |   114 | 7010 |                | 
ok |
| vm-105-disk-2   |      0 | 200 GiB |   120 | 7011 |                | 
ok |
| vm-104-disk-1   |      0 |  18 GiB |   119 | 7012 |                | 
ok |
| vm-105-disk-1   |      0 |  32 GiB |   121 | 7013 |                | 
ok |
| vm-1802-disk-2  |      0 |  42 GiB |   125 | 7014 |                | 
ok |
| vm-1801-disk-12 |      0 |  32 GiB |   123 | 7015 |                | 
ok |
| vm-103-disk-1   |      0 |  50 GiB |   126 | 7016 |                | 
ok |
| vm-102-disk-1   |      0 |  50 GiB |   127 | 7017 |                | 
ok |
| vm-106-disk-1   |      0 |  32 GiB |   128 | 7018 |                | 
ok |
+----------------------------------------------------------------------------+

[B] -------------------------

dmmmigrate.sh:

...
# If the controller is not executed on the local host, set this variable:
LS_CONTROLLERS="localhost"
export LS_CONTROLLERS
...
### Nodes ###
linstor node create --node-type Combined munchbase2 10.0.7.13
linstor node create --node-type Combined monetbase2 10.0.7.12

### Storage ###
linstor storage-pool-definition create drbdpool
linstor storage-pool create lvmthin munchbase2 drbdpool drbdpool/drbdthinpool
linstor storage-pool create lvmthin monetbase2 drbdpool drbdpool/drbdthinpool
...
### Resource: vm-1802-disk-2 ###
linstor resource-definition create --port 7014 vm-1802-disk-2
linstor resource-definition drbd-options --allow-two-primaries yes vm-1802-disk-2
linstor volume-definition create --vlmnr 0 --minor 125 vm-1802-disk-2 44040192K
linstor volume-definition set-property vm-1802-disk-2 0 OverrideVlmId vm-1802-disk-2_00
linstor volume-definition set-property vm-1802-disk-2 0 AllowLargerVolumeSize true
linstor resource create --node-id 1 --storage-pool drbdpool munchbase2 vm-1802-disk-2
linstor resource create --node-id 0 --storage-pool drbdpool monetbase2 vm-1802-disk-2


[C] ------------------

monetbase2:/etc/linstor# linstor error-reports show 5CA090AF-97F26-000000
ERROR REPORT 5CA090AF-97F26-000000

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Satellite
Version:                            0.9.4
Build ID: 
ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time:                         2019-03-29T09:40:59+00:00
Error time:                         2019-03-31 12:49:17
Node:                               monetbase2

============================================================

Reported error:
===============

Description:
     Failed to adjust DRBD resource vm-1802-disk-2

Category:                           LinStorException
Class name:                         ResourceException
Class canonical name: 
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at:                       Method 'adjustDrbd', Source file 
'DrbdLayer.java', Line #432

Error message:                      Failed to adjust DRBD resource 
vm-1802-disk-2

Error context:
     An error occurred while processing resource 'Node: 'monetbase2', Rsc: 
'vm-1802-disk-2''

Call backtrace:

     Method                                   Native Class:Line number
     adjustDrbd                               N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
     process                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
     process                                  N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
     processResourcesAndTheirSnapshots        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
     phaseDispatchDeviceHandlers              N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
     devMgrLoop                               N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
     run                                      N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
     run                                      N      java.lang.Thread:748

Caused by:
==========

Description:
     Execution of the external command 'drbdadm' failed.
Cause:
     The external command exited with error code 1.
Correction:
     - Check whether the external program is operating properly.
     - Check whether the command line is correct.
       Contact a system administrator or a developer if the command line is 
no longer valid
       for the installed version of the external program.
Additional information:
     The full command line executed was:
     drbdadm -vvv adjust vm-1802-disk-2

     The external command sent the following output data:
     drbdsetup detach --diskless 125
     drbdsetup del-minor 125
     drbdsetup new-minor vm-1802-disk-2 1000 0
     drbdsetup disconnect vm-1802-disk-2 1
     drbdsetup del-path vm-1802-disk-2 1 ipv4:10.0.7.12:7014 
ipv4:10.0.7.13:7014
     drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000 
ipv4:10.0.7.13:7000


     The external command sent the follwing error information:
      [ne] vol:0 minor differs: r=125 c=1000
     Value of 'shared-secret' differs: r=bXj5Gn1qO2YXgQoX6wLE 
c=aPDz2JoXsWxbscd2j70f
     vm-1802-disk-2: Failure: (173) Combination of local address(port) and 
remote address(port) already in use

     Command 'drbdsetup new-path vm-1802-disk-2 1 ipv4:10.0.7.12:7000 
ipv4:10.0.7.13:7000' terminated with exit code 10


Category:                           LinStorException
Class name:                         ExtCmdFailedException
Class canonical name: 
com.linbit.extproc.ExtCmdFailedException
Generated at:                       Method 'execute', Source file 
'DrbdAdm.java', Line #455

Error message:                      The external command 'drbdadm' exited 
with error code 1


Call backtrace:

     Method                                   Native Class:Line number
     execute                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
     adjust                                   N 
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
     adjustDrbd                               N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
     process                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
     process                                  N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
     processResourcesAndTheirSnapshots        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
     phaseDispatchDeviceHandlers              N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
     devMgrLoop                               N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
     run                                      N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
     run                                      N      java.lang.Thread:748


END OF ERROR REPORT.

[D] ------------------

monetbase2:/etc/linstor#  linstor error-reports show 5CA09099-17F52-000000
ERROR REPORT 5CA09099-17F52-000000

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Satellite
Version:                            0.9.4
Build ID: 
ff7acd2330eb0cbbdcfae5c3a1ec7a8eba5df3c2
Build time:                         2019-03-29T09:40:59+00:00
Error time:                         2019-03-31 12:49:17
Node:                               munchbase2

============================================================

Reported error:
===============

Description:
     Failed to adjust DRBD resource vm-1802-disk-2

Category:                           LinStorException
Class name:                         ResourceException
Class canonical name: 
com.linbit.linstor.storage.layer.exceptions.ResourceException
Generated at:                       Method 'adjustDrbd', Source file 
'DrbdLayer.java', Line #432

Error message:                      Failed to adjust DRBD resource 
vm-1802-disk-2

Error context:
     An error occurred while processing resource 'Node: 'munchbase2', Rsc: 
'vm-1802-disk-2''

Call backtrace:

     Method                                   Native Class:Line number
     adjustDrbd                               N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:432
     process                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
     process                                  N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
     processResourcesAndTheirSnapshots        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
     phaseDispatchDeviceHandlers              N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
     devMgrLoop                               N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
     run                                      N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
     run                                      N      java.lang.Thread:748

Caused by:
==========

Description:
     Execution of the external command 'drbdadm' failed.
Cause:
     The external command exited with error code 1.
Correction:
     - Check whether the external program is operating properly.
     - Check whether the command line is correct.
       Contact a system administrator or a developer if the command line is 
no longer valid
       for the installed version of the external program.
Additional information:
     The full command line executed was:
     drbdadm -vvv adjust vm-1802-disk-2

     The external command sent the following output data:
     drbdsetup new-peer vm-1802-disk-2 0 --_name=monetbase2 
--allow-two-primaries=yes --shared-secret=aPDz2JoXsWxbscd2j70f 
--cram-hmac-alg=sha1
     drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000 
ipv4:10.0.7.12:7000


     The external command sent the follwing error information:
     Value of 'rs-discard-granularity' differs: r=524288 c=65536
     vm-1802-disk-2: Failure: (173) Combination of local address(port) and 
remote address(port) already in use

     Command 'drbdsetup new-path vm-1802-disk-2 0 ipv4:10.0.7.13:7000 
ipv4:10.0.7.12:7000' terminated with exit code 10


Category:                           LinStorException
Class name:                         ExtCmdFailedException
Class canonical name: 
com.linbit.extproc.ExtCmdFailedException
Generated at:                       Method 'execute', Source file 
'DrbdAdm.java', Line #455

Error message:                      The external command 'drbdadm' exited 
with error code 1


Call backtrace:

     Method                                   Native Class:Line number
     execute                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:455
     adjust                                   N 
com.linbit.linstor.storage.layer.adapter.drbd.utils.DrbdAdm:86
     adjustDrbd                               N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:414
     process                                  N 
com.linbit.linstor.storage.layer.adapter.drbd.DrbdLayer:243
     process                                  N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:598
     processResourcesAndTheirSnapshots        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:256
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceHandlerImpl:124
     dispatchResources                        N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:244
     phaseDispatchDeviceHandlers              N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:855
     devMgrLoop                               N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:597
     run                                      N 
com.linbit.linstor.core.devmgr.DeviceManagerImpl:514
     run                                      N      java.lang.Thread:748


END OF ERROR REPORT.

[E] --------------------------
/var/lib/linstor.d/vm-1802-disk-2.res:

# This file was generated by linstor(0.9.4), do not edit manually.

resource "vm-1802-disk-2"
{
     template-file "linstor_common.conf";

     net
     {
         cram-hmac-alg     sha1;
         shared-secret     "TKoHhL4KThL165NGg/ja";
         allow-two-primaries yes;
     }

     on monetbase2
     {
         volume 0
         {
             disk        /dev/drbdpool/vm-1802-disk-2_00;
             disk
             {
                 discard-zeroes-if-aligned yes;
                 rs-discard-granularity 65536;
             }
             meta-disk   internal;
             device      minor 1000;
         }
         node-id    0;
     }

     on munchbase2
     {
         volume 0
         {
             disk        /dev/drbd/this/is/not/used;
             disk
             {
                 discard-zeroes-if-aligned yes;
                 rs-discard-granularity 65536;
             }
             meta-disk   internal;
             device      minor 1000;
         }
         node-id    1;
     }

     connection
     {
         host monetbase2 address ipv4 10.0.7.12:7000;
         host munchbase2 address ipv4 10.0.7.13:7000;
     }
}



==========================================================2019-03-31==13:56==

   Dr. Wolfgang Walkowiak     Phone: +49-271-740-3889
   Fakultaet IV / Physik      Fax  : +49-271-740-3886
   Emmy Noether Campus
   Universitaet Siegen    --> Wolfgang.Walkowiak at hep.physik.uni-siegen.de
   Walter-Flex-Str. 3         Wolfgang.Walkowiak at gmx.net
   57068 Siegen
   Germany

=============================================================================


More information about the drbd-user mailing list