<div dir="ltr"><div>Could still be related to this fix:</div><div><br></div><div> * fix timeout detection after idle periods and for configs with ko-count<br> when a disk on an a secondary stops delivering IO-completion events<br></div><div><br></div><div>So if you have a ko-count set, this should be fixed.</div><div>Or it is something completely different... ;)</div><div><br></div><div>Cheers,</div><div>Rene</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 27, 2021 at 1:25 PM Andreas Pflug <<a href="mailto:pgadmin@pse-consulting.de">pgadmin@pse-consulting.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm running a Proxmox cluster with 3 disk nodes and 3 diskless nodes<br>
with drbd 9.1.1. The disk nodes have storage on md raid6 (8 ssds each)<br>
with a journal on an optane device.<br>
<br>
Yesterday, the whole cluster was severely impacted when one node had<br>
write problems. There is no indication for any hardware problem, no<br>
events whatsoever. What happened, taken from the logs:<br>
<br>
- one diskless node reports "sending time expired" for some devices on a<br>
specific disk node. After 30 seconds, it disconnects those devices on<br>
that node.<br>
- the disk node logs state change to outdated.<br>
- After 80s, the disk node logs "task blocked for more than 120<br>
seconds". These tasks are 8 drbd_r_xxx processes, but also md2_reclaim.<br>
- No more logging after that.<br>
<br>
After that, the whole cluster was severely impacted, most vms<br>
unresponsive. The node hosts were still accessible, with no more kernel<br>
logging.<br>
<br>
After analyzing the situation, assuming a single node would block<br>
everything, that node was rebooted (no normal reboot possible, needed<br>
"echo b >/proc/sysrq-trigger"). This did help, everything back to normal.<br>
<br>
So apparently there are situations when a backing storage problem might<br>
block all drbd processing in a way that prevents normal timeout<br>
detection and subsequent disconnection on other nodes. Reading the 9.1.2<br>
release notes, this doesn't seem to be addressed there.<br>
<br>
Regards,<br>
Andreas<br>
<br>
_______________________________________________<br>
Star us on GITHUB: <a href="https://github.com/LINBIT" rel="noreferrer" target="_blank">https://github.com/LINBIT</a><br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="https://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">https://lists.linbit.com/mailman/listinfo/drbd-user</a><br>
</blockquote></div>