<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>after some debugging I solved the problem. The cause was is in
running cinder-volume on controller host, while it's required to
be on host where storage reside as well.</p>
<p>=====<br>
</p>
<p>With more details, if you run cinder-volume on controller, while
volume itself is on another hosts, driver will not find matching
record:<br>
</p>
<p>DEBUG cinder.volume.drivers.linstordrv: full resource name
"CV_15c4fa98-76a5-44d6-9701-c89a8a848b82" _return_drbd_config
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:1105</p>
<p>--- cinder-volume is on ctrl1:<br>
</p>
<p>DEBUG cinder.volume.drivers.linstordrv: _get_rsc_path called with
CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 on ctrl1 _get_rsc_path
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:425</p>
<p>--- while volume is on cmpX hosts:<br>
</p>
<p>DEBUG cinder.volume.drivers.linstordrv: found an entry with
CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 on cmp1 _get_rsc_path
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:429<br>
DEBUG cinder.volume.drivers.linstordrv: found an entry with
CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 on cmp2 _get_rsc_path
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:429<br>
DEBUG cinder.volume.drivers.linstordrv: found an entry with
CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 on cmp3 _get_rsc_path
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:429</p>
<p>--- since no match, driver returns None<br>
</p>
<p>DEBUG cinder.volume.drivers.linstordrv: resource path "None"
_return_drbd_config
/usr/lib/python3/dist-packages/cinder/volume/drivers/linstordrv.py:1107</p>
<p>and then Nova fails with "None" path to device.</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 21.05.2020 12:45, Volodymyr Litovka
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:34d4155f-3f9d-5bfa-e28b-2308ff71a44b@gmx.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<pre>Hi colleagues,
does anybody here use Linstor as backend for Openstack? I can not launch VM (synopsys below) while everything looks fine until the last step :-)
In brief about Openstack configuration:
- controllers: cinder-api, cinder-scheduler, cinder-volume (modified with latest linstor driver from git), linstor-client, python-linstor
- compute nodes: linstor-controller, linstor-satellite, linstor-client, python-linstor; node types - Combined
- cinder.conf[1]
- volume-type registered with corresponding properties[2]
I'm able to manipulate volumes - create/delete[3] and every existing volume has the corresponding linstor repr[4] and LVM LV[5] on the compute nodes
The problem is when I try to create VM using
openstack server create --flavor 2x2 --network [...] --volume 15c4fa98-76a5-44d6-9701-c89a8a848b82 fearst
Nova fails because of libvirt XML definition (note source dev='None' below):
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='None'/>
<target dev='vda' bus='virtio'/>
<serial>15c4fa98-76a5-44d6-9701-c89a8a848b82</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</disk>
I do not have another working DRBD/Linstor/Openstack installation and can't even imagine, where to look into the problem - whether it is in Nova, Cinder or Linstor configuration itself.
I will appreciate if anybody can give me some guidance on where to seek for the solution.
Thank you!
====================================
[1] cinder.conf
[DEFAULT]
enabled_backends = linstor
default_volume_type = linstor
[linstor]
storage_availability_zone = nova
volume_backend_name = linstor
volume_driver = cinder.volume.drivers.linstordrv.LinstorDrbdDriver
linstor_autoplace_count = 2
linstor_default_volume_group_name=DfltRscGrp
linstor_default_uri=linstor://runner
linstor_default_storage_pool_name=drbdpool
linstor_default_resource_size=1
linstor_volume_downsize_factor=4096
where 'runner' is virtual ip of linstor-controller, protected by pacemaker.
[2] openstack volume type show linstor
+--------------------+--------------------------------------+
| Field | Value |
+--------------------+--------------------------------------+
| access_project_ids | None |
| description | None |
| id | d2025962-503a-4f37-93bd-b766bb346a42 |
| is_public | True |
| name | linstor |
| properties | volume_backend_name='linstor' |
| qos_specs_id | None |
+--------------------+--------------------------------------+
[3] openstack volume show fearst
+--------------------------------+-----------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2020-05-21T08:42:11.000000 |
| description | None |
| encrypted | False |
| id | 15c4fa98-76a5-44d6-9701-c89a8a848b82 |
| migration_status | None |
| multiattach | False |
| name | fearst |
| os-vol-host-attr:host | ctrl1@linstor#linstor |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 7acec404393344cbabde07a22bbe6b3f |
| properties | |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| type | linstor |
| updated_at | 2020-05-21T08:42:18.000000 |
| user_id | 1048a19e94234451a61c3d9d46907ceb |
| volume_image_metadata | {'signature_verified': 'False', |
| | 'image_id': '15e992e8-a601-44cd-a2f0-8f34588a4d18',|
| | 'image_name': 'cirros', |
| | 'checksum': '1d3062cd89af34e419f7100277f38b2b', |
| | 'container_format': 'bare', |
| | 'disk_format': 'qcow2', |
| | 'min_disk': '0', |
| | 'min_ram': '0', |
| | 'size': '16338944'} |
+--------------------------------+-----------------------------------------------------+
[4] linstor r l
╭───────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════╡
┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp1 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊
┊ CV_15c4fa98-76a5-44d6-9701-c89a8a848b82 ┊ cmp2 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊
╰───────────────────────────────────────────────────────────────────────────────────╯
Note, that cmp3 is arbiter and has no 'by-disk' representation:
root@cmp1:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-disk/sds:
lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
root@cmp2:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-disk/sds:
lrwxrwxrwx 1 root root 17 May 21 11:42 CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000 -> ../../../drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
root@cmp3:~# ls -lR /dev/drbd*
brw-rw---- 1 root disk 147, 1000 May 21 11:42 /dev/drbd1000
/dev/drbd/by-res/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82:
lrwxrwxrwx 1 root root 17 May 21 11:42 0 -> ../../../drbd1000
[5] root@cmp1:~# lvdisplay
[ ... ]
--- Logical volume ---
LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
VG Name sds
LV UUID jGzcIQ-yRpH-RKxP-ExsH-JSn6-kTFn-8kpKYO
LV Write Access read/write
LV Creation host, time cmp1, 2020-05-21 11:42:14 +0300
LV Pool name thin
LV Status available
# open 2
LV Size 5.00 GiB
Mapped size 0.02%
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
root@cmp2:~# lvdisplay
[ ... ]
--- Logical volume ---
LV Path /dev/sds/CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
LV Name CV_15c4fa98-76a5-44d6-9701-c89a8a848b82_00000
VG Name sds
LV UUID 8pVwMo-e3ed-Sy0w-06IZ-M7Mh-YVIL-BtKZEG
LV Write Access read/write
LV Creation host, time cmp2, 2020-05-21 11:42:14 +0300
LV Pool name thin
LV Status available
# open 2
LV Size 5.00 GiB
Mapped size 0.02%
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison</pre>
</body>
</html>