[DRBD-user] Kernel tainted on workqueue: do_submit [drbd]

[kvaps]

Hi we have very strange issue with one volume, we are using lvm and
drbd with diskless replica.

After we start workload on diskless node it it is working fine, but
after some time all I/O is hung and we have the next messages in
dmesg:

from m8c25 (diskless node):

    [Sun Mar 10 05:42:28 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 6
    [Sun Mar 10 05:42:28 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 6
    [Sun Mar 10 05:42:34 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 5
    [Sun Mar 10 05:42:34 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 6
    [Sun Mar 10 05:42:40 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 4
    [Sun Mar 10 05:42:40 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 5
    [Sun Mar 10 05:42:46 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 3
    [Sun Mar 10 05:42:46 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 4
    [Sun Mar 10 05:42:52 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 2
    [Sun Mar 10 05:42:52 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 3
    [Sun Mar 10 05:42:59 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: [drbd_s_csi-9400/6151]
sending time expired, ko = 1
    [Sun Mar 10 05:42:59 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 2
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: conn( Connected ->
NetworkFailure ) peer( Secondary -> Unknown )
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104 m12c4: pdsk(
UpToDate -> DUnknown ) repl( Established -> Off )
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: [drbd_s_csi-9400/6154]
sending time expired, ko = 1
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: ack_receiver
terminated
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: Terminating ack_recv
thread
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104: sending new
current UUID: C07BEF398E0AD984
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: Preparing remote state
change 1414712872
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: Committing remote
state change 1414712872 (primary_nodes=8)
    [Sun Mar 10 05:43:05 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104 m12c4: pdsk(
DUnknown -> Outdated )
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m7c14: conn( Connected ->
NetworkFailure ) peer( Secondary -> Unknown )
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104 m7c14: pdsk(
UpToDate -> DUnknown ) repl( Established -> Off )
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: Connection closed
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: conn( NetworkFailure
-> Unconnected )
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: Restarting receiver
thread
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104: IO ERROR: neither
local nor remote data, sector 25237760+8
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696 m12c4: conn( Unconnected ->
Connecting )
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104: IO ERROR: neither
local nor remote data, sector 1001578856+8
    [Sun Mar 10 05:43:11 2019] drbd
csi-940049b5-d5aa-46f0-9a73-fe601c3fc696/0 drbd1104: IO ERROR: neither
local nor remote data, sector 713186240+8