<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EstiloCorreo17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 3.0cm 70.85pt 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ES link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>If bonding between the 2 nodes is active/pasive. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>You network speed is only 1Gb/s = aprox 120MB/s<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>De:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> drbd-user-bounces@lists.linbit.com [mailto:drbd-user-bounces@lists.linbit.com] <b>En nombre de </b>Max Weissbach<br><b>Enviado el:</b> lunes, 23 de febrero de 2015 17:05<br><b>Para:</b> drbd-user@lists.linbit.com<br><b>Asunto:</b> [DRBD-user] DRBD sync speed much slower than expected<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi,<br><br>I'm using drbd (8.3.13) in a ganeti 2.11.6 environment.<br><br>The system is set up as follows:<br>* 3 physical machines, 2x 6 core xeon + 96GB RAM each node<br>(all hosts running Debian Wheezy)<br>* 4 SSD @ RAID 10 via hardware raid controller<br>* 500 GByte of this volume as an lvm vg for ganeti (as required for ganeti)<br>* network between the nodes 2x 1Gbit using port bonding (active/slave failover, bond mode 0)<br>* network speed, though is limited to 500 Mbit/s by the DC (money and stuff -.-*)<br><br>My problem is the following. The first, lets say, 2 weeks of setting up the ganeti cluster (and filling it with kvm VMs), everything used to be fine.<br>DRBD Volumes were syncing at top speeds as seen in iftop monitoring the bonding device, which means 400-450 Mbits - no complains.<br>Then suddenly every installation took much longer than it has recently. Syncing speeds never raised 130 Mbits from that moment on.<br><br>I started searching at the bottom, so network and physical disks. Network had the desired capacity all the time, with scp running 3 or 4 times faster than the drbd sync<br>(even WHILE having a sync in progress). iperf measurements from one node to another confirmed that:<br><br>#################<br><br>Client connecting to 10.46.0.2, TCP port 5001<br>TCP window size: 64.0 KByte (default)<br>------------------------------<o:p></o:p></p><div><p class=MsoNormal>------------------------------<br>[ 3] local 10.46.0.3 port 47716 connected with 10.46.0.2 port 5001<br>[ ID] Interval Transfer Bandwidth<br>[ 3] 0.0-10.0 sec 520 MBytes 435 Mbits/sec<br><br>#################<br><br>The disks are okay too, could not imagine a raid10 ssd raid hitting the break to 100 Mbit/s anyways, dd to be sure:<br><br>#################<br><br># dd if=/dev/zero of=/tmp/foobar bs=1k count=1000000<br>1000000+0 Datensätze ein<br>1000000+0 Datensätze aus<br>1024000000 Bytes (1,0 GB) kopiert, 1,96102 s, 522 MB/s<br><br>#################<br><br>I've then tried to dig a bit deeper into drbd and made a couple of settings there.<br><br>e.g.<br>* trying different sync-rate (in static mode)<br>* trying dynamic syncer with several min and max limits<br><br>When enabling a ganeti specific option --prealloc-disk-wipe the behaviour recovered for a short time but speeds dropped again :/<br><br>I refocused the network, and set up MTU 9000 and txqueuelen 10000 for the bonding interface which again fixed it for a few minutes just to get bad again.<br><br>I've tried to set several tcp related sysctl options such as net.core.netdev_max_backlog and net.ipv4.tcp_mtu_probing.<br><br>It all ends in syncing speed with max 100 mbit/s.<br><br>My problem isn't the initial creation of VMs (initial sync could be put into background so the vm can already install while the sync is in progress). The main point ist that the slow sync is acts as a bottleneck for disk actions on the vms themself.<br><br>That gives me technically a limit of 100 Mbit/s virtual drive speed (especially when handling with big files). I could use drbd Protocol A to prevent this but data integrity in case of failures is more important to me.<br><br>Actually I'm running out of ideas what else to try.<br><br>Anyone an idea where I forgot to look - any suggestions?<br><br>Best regards and thanks in advance<br>Max<o:p></o:p></p></div></div></div></body></html>