[DRBD-user] Resource is 'Blocked: upper'

[Julien Escario]

Le 27/08/2018 à 18:15, Julien Escario a écrit :
> Le 27/08/2018 à 17:44, Lars Ellenberg a écrit :
>> On Mon, Aug 27, 2018 at 05:01:52PM +0200, Julien Escario wrote:
>>> Hello,
>>> We're stuck in a strange situation. One of our ressources is marked as :
>>> volume 0 (/dev/drbd155): UpToDate(normal disk state) Blocked: upper
>>>
>>> I used drbdtop to get this info because drbdadm hangs.
>>>
>>> I can also see a drbdsetup process blocked :
>>> drbdsetup disk-options 155 --set-defaults --read-balancing=prefer-local
>>> --al-extents=6481 --al-updates=no --md-flushes=no
>>
>> maybe check /proc/<pid>/stack to see where it blocks.
>> Also, what DRBD (kernel) version is this?
> 
> # cat /proc/drbd
> version: 9.0.12-1 (api:2/proto:86-112)
> GIT-hash: 7eb4aef4abbfba8ebb1afbcc30574df74db0063e build by root at vm9, 2018-03-27
> 15:55:44
> Transports (api:16): tcp (9.0.12-1)
> 
> 
> # cat /proc/22538/stack
> [<ffffffffc0b6dc67>] drbd_al_shrink+0xd7/0x1a0 [drbd]
> [<ffffffffc0b85712>] drbd_adm_disk_opts+0x2b2/0x580 [drbd]
> [<ffffffff8f028d83>] genl_family_rcv_msg+0x203/0x3f0
> [<ffffffff8f028fbc>] genl_rcv_msg+0x4c/0x90
> [<ffffffff8f02846c>] netlink_rcv_skb+0xec/0x120
> [<ffffffff8f028b68>] genl_rcv+0x28/0x40
> [<ffffffff8f027be2>] netlink_unicast+0x192/0x230
> [<ffffffff8f027f52>] netlink_sendmsg+0x2d2/0x3c0
> [<ffffffff8efcbfde>] sock_sendmsg+0x3e/0x50
> [<ffffffff8efcc075>] sock_write_iter+0x85/0xf0
> [<ffffffff8ea555c7>] new_sync_write+0xe7/0x140
> [<ffffffff8ea55649>] __vfs_write+0x29/0x40
> [<ffffffff8ea563b5>] vfs_write+0xb5/0x1a0
> [<ffffffff8ea57a75>] SyS_write+0x55/0xc0
> [<ffffffff8f2001a1>] entry_SYSCALL_64_fastpath+0x24/0xab
> [<ffffffffffffffff>] 0xffffffffffffffff

Just got another interesting one, perhaps more at the root of the problem :
Process is named [drbd_r_vm-102-d]. So kernel receiver for vm-102 resource.

his stack file is :
# cat /proc/12024/stack
[<ffffffffc0b64f17>] conn_disconnect+0x667/0x7f0 [drbd]
[<ffffffffc0b65dbc>] drbd_receiver+0x1cc/0x690 [drbd]
[<ffffffffc0b74e20>] drbd_thread_setup+0x70/0x160 [drbd]
[<ffffffff8e8a886c>] kthread+0x10c/0x140
[<ffffffff8f200485>] ret_from_fork+0x35/0x40
[<ffffffffffffffff>] 0xffffffffffffffff

What is this processus waiting for ? No clue actually.

I'm trying to find a way to 'unblock' this process (by giving it what he waits
for) but perhaps is it impossible.

I need more in-depth kernel knowledge.

Julien