[DRBD-user] linstor-satellite connection issues after upgrade
Volodymyr Litovka
doka.ua at gmx.com
Tue May 12 17:55:15 CEST 2020
Hi Gábor,
please see inline
On 12.05.2020 08:44, Gábor Hernádi wrote:
> If you like you can poke it a little using "node list", "storage-pool
> list" or even "node reconnect $nodeName" during the trace logging
I found how to reproduce the issue: after 'systemctl restart
linstor-satellite' on any node (e.g. cmp2 in my case)
=== Initial state
[OSL] root at cmp2:~# linstor node list
╭───────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═══════════════════════════════════════════════════════╡
┊ cmp1 ┊ COMBINED ┊ 172.16.16.201:3366 (PLAIN) ┊ Online ┊
┊ cmp2 ┊ COMBINED ┊ 172.16.16.202:3366 (PLAIN) ┊ Online ┊
┊ cmp3 ┊ COMBINED ┊ 172.16.16.203:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────╯
[OSL] root at cmp2:~# linstor sp l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
=== Then I restart service using systemctl and controller reconnects to
satellite
[OSL] root at cmp2:~# date; systemctl restart linstor-satellite
Tue May 12 18:40:46 EEST 2020
[OSL] root at cmp2:~# date; linstor sp l
Tue May 12 18:40:51 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ ┊ ┊ True ┊ Warning ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root at cmp2:~# date; linstor sp l
Tue May 12 18:41:11 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Note that DfltDisklessStorPool remains in warning state. If I issue
"node reconnect" immediately, nothing happens; if I issue "node
reconnect" in few minutes, everything becomes "Ok". Note 'date' in
snippets below - warning gone after 2nd reconnect in four minutes after
service restart:
[OSL] root at cmp2:~# date; linstor node reconnect cmp2
Tue May 12 18:41:22 EEST 2020
SUCCESS:
Nodes [cmp2] will be reconnected.
[OSL] root at cmp2:~# date; linstor sp l
Tue May 12 18:41:24 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root at cmp2:~# date; linstor sp l
Tue May 12 18:43:59 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root at cmp2:~# date; linstor node reconnect cmp2
Tue May 12 18:44:02 EEST 2020
SUCCESS:
Nodes [cmp2] will be reconnected.
[OSL] root at cmp2:~# date; linstor sp l
Tue May 12 18:44:03 EEST 2020
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
> Anyways, I would like to ask you for trace logs of the controller (and
> maybe also from a satellite if you can catch it while it has
> connection-issues)
I will appreciate if you can provide instructions on how to enable trace
logs.
Thank you.
--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linbit.com/pipermail/drbd-user/attachments/20200512/c9e15470/attachment-0001.htm>
More information about the drbd-user
mailing list