<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Gábor,</p>
<p>please see inline<br>
</p>
<div class="moz-cite-prefix">On 12.05.2020 08:44, Gábor Hernádi
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAL16tNZsSfrxKuZMefQ0avx75jtC9vp33JoqKdAYmCq5orLoVw@mail.gmail.com">
<div dir="ltr">
<div>If you like you can poke it a little using "node list",
"storage-pool list" or even "node reconnect $nodeName" during
the trace logging</div>
</div>
</blockquote>
<p>I found how to reproduce the issue: after 'systemctl restart
linstor-satellite' on any node (e.g. cmp2 in my case)</p>
<p>=== Initial state</p>
<pre>[OSL] root@cmp2:~# linstor node list
╭───────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═══════════════════════════════════════════════════════╡
┊ cmp1 ┊ COMBINED ┊ 172.16.16.201:3366 (PLAIN) ┊ Online ┊
┊ cmp2 ┊ COMBINED ┊ 172.16.16.202:3366 (PLAIN) ┊ Online ┊
┊ cmp3 ┊ COMBINED ┊ 172.16.16.203:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────╯
[OSL] root@cmp2:~# linstor sp l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
</pre>
<p>=== Then I restart service using systemctl and controller
reconnects to satellite<br>
</p>
<pre>[OSL] root@cmp2:~# date; systemctl restart linstor-satellite
Tue May 12 18:40:46 EEST 2020
[OSL] root@cmp2:~# date; linstor sp l
Tue May 12 18:40:51 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ ┊ ┊ True ┊ Warning ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root@cmp2:~# date; linstor sp l
Tue May 12 18:41:11 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
</pre>
<p>Note that DfltDisklessStorPool remains in warning state. If I
issue "node reconnect" immediately, nothing happens; if I issue
"node reconnect" in few minutes, everything becomes "Ok". Note
'date' in snippets below - warning gone after 2nd reconnect in
four minutes after service restart:<br>
</p>
<pre>[OSL] root@cmp2:~# date; linstor node reconnect cmp2
Tue May 12 18:41:22 EEST 2020
SUCCESS:
Nodes [cmp2] will be reconnected.
[OSL] root@cmp2:~# date; linstor sp l
Tue May 12 18:41:24 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root@cmp2:~# date; linstor sp l
Tue May 12 18:43:59 EEST 2020
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Warning ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
[OSL] root@cmp2:~# date; linstor node reconnect cmp2
Tue May 12 18:44:02 EEST 2020
SUCCESS:
Nodes [cmp2] will be reconnected.
[OSL] root@cmp2:~# date; linstor sp l
Tue May 12 18:44:03 EEST 2020
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ cmp1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ DfltDisklessStorPool ┊ cmp3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊
┊ drbdpool ┊ cmp1 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp2 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
┊ drbdpool ┊ cmp3 ┊ LVM_THIN ┊ sds/thin ┊ 93 GiB ┊ 93 GiB ┊ True ┊ Ok ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
</pre>
<p>
<blockquote type="cite">Anyways, I would like to ask you for trace
logs of the controller (and maybe also from a satellite if you
can catch it while it has connection-issues)</blockquote>
I will appreciate if you can provide instructions on how to enable
trace logs.<br>
</p>
<p>Thank you.</p>
<pre class="moz-signature" cols="72">--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison</pre>
</body>
</html>