<div dir="ltr">Dear Igor,<br><br>Thank you for your reply.<br>Seems that i managed to sort it out.<div><br></div><div>In the end, i have created a script that was monitoring the logs and then doing several checks when the logs were generated.<br>Turns out that the issue was memory-related.</div><div><br>Basically, seems that the IPoIB module needed n*64kb memory pages available to work while there were none, because the customer had computational jobs running on the head node.<br>I could see that with iperf, the BW was around 100kb/s.<br><br>i have enabled the vm.reclaim_zone variable, so it will be able to reclaim the cached memory. It will probably impact on the performances, but at least the DRBD is stable.<br><br>It's been 2 days without errors :)<br><br>Regards,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 8 June 2017 at 02:00, Igor Cicimov <span dir="ltr"><<a href="mailto:icicimov@gmail.com" target="_blank">icicimov@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><br><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On 8 Jun 2017 9:40 am, "Igor Cicimov" <<a href="mailto:icicimov@gmail.com" target="_blank">icicimov@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_4427131662768840725quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote"><div class="m_4427131662768840725elided-text">On 6 Jun 2017 7:23 pm, "Andrea del Monaco" <<a href="mailto:andrea.delmonaco@clustervision.com" target="_blank">andrea.delmonaco@clustervisio<wbr>n.com</a>> wrote:<br type="attribution"></div><blockquote class="m_4427131662768840725m_-5336911155753227896quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_4427131662768840725elided-text"><div dir="ltr">Hello everybody, <div><br></div><div>I am currently facing some issues with the DRBD syncronization.</div><div>Here is the config file:<br><div>global {<br></div><div> usage-count no;</div><div>}</div><div><br></div><div>common {</div><div> startup {</div><div> wfc-timeout 15;</div><div> degr-wfc-timeout 15;</div><div> outdated-wfc-timeout 15;</div><div> }</div><div> disk {</div><div> resync-rate 80M;</div><div> disk-flushes no;</div><div> disk-barrier no;</div><div> al-extents 3389;</div><div> c-fill-target 0;</div><div> c-plan-ahead 18;</div><div> c-max-rate 200M;</div><div> }</div><div> net {</div><div> protocol C;</div><div> max-buffers 8000;</div><div> max-epoch-size 8000;</div><div> sndbuf-size 1024k;</div><div> }</div><div>}</div><div><br></div><div>resource cmshareddrbdres {</div><div> net {</div><div> cram-hmac-alg sha1;</div><div> shared-secret xxxxxxx;</div><div> after-sb-0pri discard-younger-primary;</div><div> after-sb-1pri discard-secondary;</div><div> csums-alg md5;</div><div> }</div><div> on master1 {</div><div> device /dev/drbd1;</div><div> disk /dev/sdb;</div><div> address <a href="http://10.149.255.254:7789" target="_blank">10.149.255.254:7789</a>;</div><div> meta-disk internal;</div><div> }</div><div> on master2 {</div><div> device /dev/drbd1;</div><div> disk /dev/sdb;</div><div> address <a href="http://10.149.255.253:7789" target="_blank">10.149.255.253:7789</a>;</div><div> meta-disk internal;</div></div><div><div> }</div><div>}<br><br>The network <a href="http://10.149.0.0/16" target="_blank">10.149.0.0/16</a> is using IPoIB.<br><br>The messages that i see are (first master): <a href="https://pastebin.com/0xCLceeD" target="_blank">https://pastebin.com/<wbr>0xCLceeD</a></div><div><br></div><div>Suspect messages:<br>[Sun Jun 4 03:50:17 2017] block drbd1: logical block size of local backend does not match (drbd:512, backend:4096); was this a late attach?</div><div>[Sun Jun 4 03:51:01 2017] drbd cmshareddrbdres: [drbd_w_cmshared/3640] sock_sendmsg time expired, ko = 6</div><div>[Sun Jun 4 03:34:12 2017] block drbd1: We did not send a P_BARRIER for 84203ms > ko-count (7) * timeout (60 * 0.1s); drbd kernel thread blocked?<br>(I see so many of these)<br><br>To me, i would say that there is some issue with the network, but i am not sure, because in that case i would expect drbd to be able to send the messages but going in timeout on the other side.</div><div><br></div><div>I have tried to stress it and i couldn't reproduce it, so it doesn't seem to be load-related.</div><div><br></div><div><div>[root@master1 ~]# uname -r</div><div>3.10.0-327.el7.x86_64</div><div>[root@master1 ~]# rpm -qa | grep drbd</div><div>kmod-drbd84-8.4.7-1_1.el7.elre<wbr>po.x86_64</div><div>drbd84-utils-8.9.5-1.el7.elrep<wbr>o.x86_64</div></div><div><br></div><div>Any ideas?<br><br><br>Regards,</div></div><div>-- <br><div class="m_4427131662768840725m_-5336911155753227896m_56011209079771550gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br><table height="195" width="312" style="font-size:7pt;font-family:Tahoma,Arial,Helvetica;padding:0px;border:1px solid rgb(234,239,242)"><tbody><tr valign="top"><td colspan="2"><img alt="clustervision_logo.png" title="" src="http://www.clustervision.com/images/cv_sig.gif"></td></tr><tr><td valign="bottom" nowrap style="padding-left:12px"><font style="font-size:9pt;font-weight:bold">Andrea Del Monaco<br></font><font style="font-size:7pt">Internal Engineer<br> <br> <br>Mob: +31 64 166 4003<br>Skype: delmonaco.andrea<br><a href="mailto:andrea.delmonaco@clustervision.com" style="text-decoration:none;color:rgb(57,136,194)" target="_blank">andrea.delmonaco@clustervision<wbr>.com</a></font><br> <br></td><td valign="bottom" nowrap><font style="font-size:8pt;font-weight:bold">ClusterVision BV<br></font><font style="font-size:7pt">Gyroscoopweg 56<br>1042 AC Amsterdam<br>The Netherlands<br>Tel: <a href="tel:+31%2020%20407%207550" value="+31204077550" target="_blank">+31 20 407 7550</a><br>Fax: <a href="tel:+31%2084%20759%208389" value="+31847598389" target="_blank">+31 84 759 8389</a><br><a href="http://www.clustervision.com/" style="text-decoration:none;color:rgb(0,63,119)" target="_blank">www.clustervision.com</a></font><br> <br></td></tr></tbody></table></div></div></div></div></div></div></div></div></div>
</div></div>
<br></div>______________________________<wbr>_________________<br>
drbd-user mailing list<br>
<a href="mailto:drbd-user@lists.linbit.com" target="_blank">drbd-user@lists.linbit.com</a><br>
<a href="http://lists.linbit.com/mailman/listinfo/drbd-user" rel="noreferrer" target="_blank">http://lists.linbit.com/mailma<wbr>n/listinfo/drbd-user</a><br>
<br></blockquote></div>The ko-count thing from the log means the secondary fails to commit the writes in expected time frame which looks to me like backing device storage/driver/os issues rather than drbd. I would check if that works properly first if I was you.</div></div></div>
</blockquote></div></div></div>Then test the network speed (if you havent done so already) timeout of 7x6=42sec is way too high for infiniband for this kind of issues. Bu the way, there is a Linbit technical guide for ipoib which i hope you did consult.</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><br><table height="195" width="312" style="font-size:7pt;font-family:Tahoma,Arial,Helvetica;padding:0px;border:1px solid rgb(234,239,242)"><tbody><tr valign="top"><td colspan="2"><img alt="clustervision_logo.png" title="" src="http://www.clustervision.com/images/cv_sig.gif"></td></tr><tr><td valign="bottom" nowrap style="padding-left:12px"><font style="font-size:9pt;font-weight:bold">Andrea Del Monaco<br></font><font style="font-size:7pt">Internal Engineer<br> <br> <br>Mob: +31 64 166 4003<br>Skype: delmonaco.andrea<br><a href="mailto:andrea.delmonaco@clustervision.com" style="text-decoration:none;color:rgb(57,136,194)" target="_blank">andrea.delmonaco@clustervision.com</a></font><br> <br></td><td valign="bottom" nowrap><font style="font-size:8pt;font-weight:bold">ClusterVision BV<br></font><font style="font-size:7pt">Gyroscoopweg 56<br>1042 AC Amsterdam<br>The Netherlands<br>Tel: +31 20 407 7550<br>Fax: +31 84 759 8389<br><a href="http://www.clustervision.com/" style="text-decoration:none;color:rgb(0,63,119)" target="_blank">www.clustervision.com</a></font><br> <br></td></tr></tbody></table></div></div></div></div></div></div></div></div></div>
</div>