<div dir="ltr"><div>Juergen,</div><div><br></div><div>Thanks for the tip! I'll do some experimentation the next chance I get.</div><div><br></div><div>I agree that the root cause seems like a performance bug from my perspective.<br></div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div><br></div>-Chris</div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 5, 2019 at 4:27 AM Juergen Sauer <<a href="mailto:juergen.sauer@automatix.de">juergen.sauer@automatix.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Am 05.02.19 um 07:49 schrieb Chris Hartman:<br>
> Greetings!<br>
> <br>
> I actually messaged about this several months(?) ago, though you<br>
> articulated it better than I.<br>
> <br>
> I run a 2-node HA VM Cluster with KVM/pacemaker on top of DRBD 8.4 very<br>
> comparable to your hardware.and have experienced similar symptoms during<br>
> backup procedures. When it's really bad, one node will fence the other<br>
> because the remote disk becomes unresponsive past the DRBD timeout<br>
> threshold (auto calculated around 42 seconds).<br>
> <br>
> My only work around has been to keep all VMs on a single node at a time and<br>
> manually move all nodes periodically- this setup tolerates the I/O spike<br>
> much better. However, we don't get the performance benefit of having both<br>
> nodes active, not to mention the added administrative overhead.<br>
<br>
Hi Chris,<br>
I found, that thotteling, limiting the I/O for virtual machine drives helps:<br>
<br>
I.E.<br>
# virsh blkdeviotune Virt-Name vda --total-iops-sec 1000<br>
--total-bytes-sec 52428800<br>
<br>
See also:<br>
<br>
<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-blockio-techniques" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_tuning_and_optimization_guide/sect-virtualization_tuning_optimization_guide-blockio-techniques</a><br>
<br>
But this is only a workaround, not the solution that drbd8 fails on<br>
heavy I/O, due self generated Load escalation.<br>
<br>
mit freundlichen Grüßen<br>
Jürgen Sauer<br>
-- <br>
Jürgen Sauer - automatiX GmbH,<br>
+49-4209-4699, <a href="mailto:juergen.sauer@automatix.de" target="_blank">juergen.sauer@automatix.de</a><br>
Geschäftsführer: Jürgen Sauer,<br>
Gerichtstand: Amtsgericht Walsrode • HRB 120986<br>
Ust-Id: DE191468481 • St.Nr.: 36/211/08000<br>
GPG Public Key zur Signaturprüfung:<br>
<a href="http://www.automatix.de/juergen_sauer_publickey.gpg" rel="noreferrer" target="_blank">http://www.automatix.de/juergen_sauer_publickey.gpg</a><br>
</blockquote></div>